Visible light image with edge marking for enhancing ir imagery

ABSTRACT

The invention relates generally to edge detection and presentation in thermal images. Infrared and visible light images comprising at least partially overlapping target scenes are analyzed. An edge detection process is performed on the visible light image to determine which pixels represent edges in the target scene. A display image is generated in which some pixels include infrared image data and in which pixels corresponding to edges in the visible light image include a predetermined color and do not include corresponding infrared image data to emphasize edges. Edge pixels in the display image can include exclusively the predetermined color, or in some examples, a blend of a predetermined color and visible light image data. Methods can include replacing one or the other of visible light edge pixels or corresponding infrared pixels with the predetermined color before combining the visible light and infrared image data to create a display image.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 15/976,299, filed May 10, 2018 and titled “VISIBLELIGHT IMAGE WITH EDGE MARKING FOR ENHANCING IR IMAGERY” which is acontinuation application of U.S. patent application Ser. No. 14/222,153,filed Mar. 21, 2014 and titled “VISIBLE LIGHT IMAGE WITH EDGE MARKINGFOR ENHANCING IR IMAGERY.” The entire content of these applications areincorporated herein by reference.

BACKGROUND

Thermal imaging cameras are used in a variety of situations. Forexample, thermal imaging cameras are often used during maintenanceinspections to thermally inspect equipment. Example equipment mayinclude rotating machinery, electrical panels, or rows of circuitbreakers, among other types of equipment. Thermal inspections can detectequipment hot spots such as overheating machinery or electricalcomponents, helping to ensure timely repair or replacement of theoverheating equipment before a more significant problem develops.

Depending on the configuration of the camera, the thermal imaging cameramay also generate a visible light image of the same object. The cameramay display the infrared image and the visible light image in acoordinated manner, for example, to help an operator interpret thethermal image generated by the thermal imaging camera. Unlike visiblelight images which generally provide good contrast between differentobjects, it is often difficult to recognize and distinguish differentfeatures in a thermal image as compared to the real-world scene. Forthis reason, an operator may rely on a visible light image to helpinterpret and focus the thermal image. For example, overlapping and/orcombining the visible light image and the thermal image can provide someguidance for the operator. However, in some situations, it can still bedifficult to distinguish edges and boundaries of objects in the thermalimage.

SUMMARY

Aspects of the present invention are directed towards edge detection anddisplay in thermal images. Embodiments of the invention can include anon-transitory computer-readable medium with instructions for causing aprocessor to perform a method for generating a display image withemphasized edges. For example, a processor can receive visible light(VL) image data comprising a plurality of VL pixels and infrared (IR)image data comprising a plurality of IR pixels. One or more IR pixelscan comprise one or more corresponding VL pixels.

The processor can analyze VL image data of the target scene and detectedges within the VL image data. Edge detection can be performed in anyof a variety of ways. The processor can determine which VL pixelscorrespond to detected edges and consider such pixels to be VL edgepixels. The processor can further generate a display image having aplurality of display pixels corresponding to one or more VL pixels.Display pixels can generally comprise VL, IR, or a combination of VL andIR pixels.

Display pixels can be generated such that some of the display pixelsinclude IR image data associated with corresponding IR pixels. Suchpixels can be presented as a blend of VL and IR pixels, for example, orinclude exclusively IR image data. Display pixels corresponding to VLedge pixels can be include a predetermined color and not include IRimage data associated with corresponding IR pixels in order to emphasizethe location of edges in the display image data. Assigning the displaypixel a predetermined color can be performed in a plurality of ways. Forexample, generating display image data can include replacing all VL edgepixels with pixels of the predetermined color, and displaying suchpixels comprising the predetermined color as the corresponding displaypixels. In other embodiments, IR pixels corresponding to VL edge pixelscan be replaced with the predetermined color to create modified IR imagedata. Subsequent blending of IR image data and VL image data can resultin display pixels corresponding to VL edge pixels comprising thepredetermined color blended with VL image data.

In some embodiments, display image data can be generated substantiallyin real time and presented on a display. Accordingly, embodiments of thepresent invention can include a display system capable of any or all ofreceiving VL and corresponding IR images, detecting edge pixels in theVL image data and generating and displaying display image in which edgesare emphasized, providing added context to an IR image. Such displaysystems can include a thermal imaging camera comprising IR and VL cameramodules configured to detect IR and VL images of a target scene,respectively.

Various aspects of the present invention can be modified by a user. Forexample, in some embodiments, a user can manipulate the edge detectionprocess so as to detect more or fewer edges. Such manipulation caninclude adjusting an edge detection sensitivity threshold used in theedge detection process. A user can adjust an amount of blending betweenVL and IR image data in creating one or more display pixels. In someembodiments, the user can select the predetermined color for customizedemphasis of edges in an IR image. Accordingly, aspects of the inventioncan include receiving one or more inputs from the user interface inorder to perform edge detection and/or display in a user-defined way.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective front view of an example thermal imaging camera.

FIG. 2 is a perspective back view of the example thermal imaging cameraof FIG. 1.

FIG. 3 is a functional block diagram illustrating example components ofthe thermal imaging camera of FIGS. 1 and 2.

FIG. 4 is a process-flow diagram illustrating a general method forincorporating detected edge information into a display image includinginfrared data.

FIG. 5 is a process-flow diagram illustrating exemplary operation ofgenerating and modifying a display image.

FIGS. 6A-6F illustrate various display images of a target scene.

DETAILED DESCRIPTION

The following detailed description is exemplary in nature and is notintended to limit the scope, applicability, or configuration of theinvention in any way. Rather, the following description provides somepractical illustrations for implementing various embodiments of thepresent invention. Examples of constructions, materials, dimensions, andmanufacturing processes are provided for selected elements, and allother elements employ that which is known to those of ordinary skill inthe field of the invention. Those skilled in the art will recognize thatmany of the noted examples have a variety of suitable alternatives.

A thermal imaging camera may be used to detect heat patterns across ascene, including an object or objects, under observation. The thermalimaging camera may detect infrared radiation given off by the scene andconvert the infrared radiation into an infrared image indicative of theheat patterns. In some embodiments, the thermal imaging camera may alsocapture visible light from the scene and convert the visible light intoa visible light image. Depending on the configuration of the thermalimaging camera, the camera may include infrared optics to focus theinfrared radiation on an infrared sensor and visible light optics tofocus the visible light on a visible light sensor.

Various embodiments provide methods and systems for producing thermalimages with reduced noise using averaging techniques. To further improveimage quality and eliminate problems that may arise from averaging (e.g.blurring, ghosting, etc.), an image alignment process is performed onthe thermal images prior to averaging.

FIGS. 1 and 2 show front and back perspective views, respectively of anexample thermal imaging camera 100, which includes a housing 102, aninfrared lens assembly 104, a visible light lens assembly 106, a display108, a laser 110, and a trigger control 112. Housing 102 houses thevarious components of thermal imaging camera 100. The bottom portion ofthermal imaging camera 100 includes a carrying handle for holding andoperating the camera via one hand. Infrared lens assembly 104 receivesinfrared radiation from a scene and focuses the radiation on an infraredsensor for generating an infrared image of a scene. Visible light lensassembly 106 receives visible light from a scene and focuses the visiblelight on a visible light sensor for generating a visible light image ofthe same scene. Thermal imaging camera 100 captures the visible lightimage and/or the infrared image in response to depressing triggercontrol 112. In addition, thermal imaging camera 100 controls display108 to display the infrared image and the visible light image generatedby the camera, e.g., to help an operator thermally inspect a scene.Thermal imaging camera 100 may also include a focus mechanism coupled toinfrared lens assembly 104 that is configured to move at least one lensof the infrared lens assembly so as to adjust the focus of an infraredimage generated by the thermal imaging camera.

In operation, thermal imaging camera 100 detects heat patterns in ascene by receiving energy emitted in the infrared-wavelength spectrumfrom the scene and processing the infrared energy to generate a thermalimage. Thermal imaging camera 100 may also generate a visible lightimage of the same scene by receiving energy in the visiblelight-wavelength spectrum and processing the visible light energy togenerate a visible light image. As described in greater detail below,thermal imaging camera 100 may include an infrared camera module that isconfigured to capture an infrared image of the scene and a visible lightcamera module that is configured to capture a visible light image of thesame scene. The infrared camera module may receive infrared radiationprojected through infrared lens assembly 104 and generate therefrominfrared image data. The visible light camera module may receive lightprojected through visible light lens assembly 106 and generate therefromvisible light data.

In some examples, thermal imaging camera 100 collects or captures theinfrared energy and visible light energy substantially simultaneously(e.g., at the same time) so that the visible light image and theinfrared image generated by the camera are of the same scene atsubstantially the same time. In these examples, the infrared imagegenerated by thermal imaging camera 100 is indicative of localizedtemperatures within the scene at a particular period of time while thevisible light image generated by the camera is indicative of the samescene at the same period of time. In other examples, thermal imagingcamera may capture infrared energy and visible light energy from a sceneat different periods of time.

Visible light lens assembly 106 includes at least one lens that focusesvisible light energy on a visible light sensor for generating a visiblelight image. Visible light lens assembly 106 defines a visible lightoptical axis which passes through the center of curvature of the atleast one lens of the assembly. Visible light energy projects through afront of the lens and focuses on an opposite side of the lens. Visiblelight lens assembly 106 can include a single lens or a plurality oflenses (e.g., two, three, or more lenses) arranged in series. Inaddition, visible light lens assembly 106 can have a fixed focus or caninclude a focus adjustment mechanism for changing the focus of thevisible light optics. In examples in which visible light lens assembly106 includes a focus adjustment mechanism, the focus adjustmentmechanism may be a manual adjustment mechanism or an automaticadjustment mechanism.

Infrared lens assembly 104 also includes at least one lens that focusesinfrared energy on an infrared sensor for generating a thermal image.Infrared lens assembly 104 defines an infrared optical axis which passesthrough the center of curvature of lens of the assembly. Duringoperation, infrared energy is directed through the front of the lens andfocused on an opposite side of the lens. Infrared lens assembly 104 caninclude a single lens or a plurality of lenses (e.g., two, three, ormore lenses), which may be arranged in series.

As briefly described above, thermal imaging camera 100 includes a focusmechanism for adjusting the focus of an infrared image captured by thecamera. In the example shown in FIGS. 1 and 2, thermal imaging camera100 includes focus ring 114. Focus ring 114 is operatively coupled(e.g., mechanically and/or electrically coupled) to at least one lens ofinfrared lens assembly 104 and configured to move the at least one lensto various focus positions so as to focus the infrared image captured bythermal imaging camera 100. Focus ring 114 may be manually rotated aboutat least a portion of housing 102 so as to move the at least one lens towhich the focus ring is operatively coupled. In some examples, focusring 114 is also operatively coupled to display 108 such that rotationof focus ring 114 causes at least a portion of a visible light image andat least a portion of an infrared image concurrently displayed ondisplay 108 to move relative to one another. In different examples,thermal imaging camera 100 may include a manual focus adjustmentmechanism that is implemented in a configuration other than focus ring114, or may, in other embodiments, simply maintain a fixed focus.

In some examples, thermal imaging camera 100 may include anautomatically adjusting focus mechanism in addition to or in lieu of amanually adjusting focus mechanism. An automatically adjusting focusmechanism may be operatively coupled to at least one lens of infraredlens assembly 104 and configured to automatically move the at least onelens to various focus positions, e.g., in response to instructions fromthermal imaging camera 100. In one application of such an example,thermal imaging camera 100 may use laser 110 to electronically measure adistance between an object in a target scene and the camera, referred toas the distance-to-target. Thermal imaging camera 100 may then controlthe automatically adjusting focus mechanism to move the at least onelens of infrared lens assembly 104 to a focus position that correspondsto the distance-to-target data determined by thermal imaging camera 100.The focus position may correspond to the distance-to-target data in thatthe focus position may be configured to place the object in the targetscene at the determined distance in focus. In some examples, the focusposition set by the automatically adjusting focus mechanism may bemanually overridden by an operator, e.g., by rotating focus ring 114.

During operation of thermal imaging camera 100, an operator may wish toview a thermal image of a scene and/or a visible light image of the samescene generated by the camera. For this reason, thermal imaging camera100 may include a display. In the examples of FIGS. 1 and 2, thermalimaging camera 100 includes display 108, which is located on the back ofhousing 102 opposite infrared lens assembly 104 and visible light lensassembly 106. Display 108 may be configured to display a visible lightimage, an infrared image, and/or a combined image that includes asimultaneous display of the visible light image and the infrared image.In different examples, display 108 may be remote (e.g., separate) frominfrared lens assembly 104 and visible light lens assembly 106 ofthermal imaging camera 100, or display 108 may be in a different spatialarrangement relative to infrared lens assembly 104 and/or visible lightlens assembly 106. Therefore, although display 108 is shown behindinfrared lens assembly 104 and visible light lens assembly 106 in FIG.2, other locations for display 108 are possible.

Thermal imaging camera 100 can include a variety of user input media forcontrolling the operation of the camera and adjusting different settingsof the camera. Example control functions may include adjusting the focusof the infrared and/or visible light optics, opening/closing a shutter,capturing an infrared and/or visible light image, or the like. In theexample of FIGS. 1 and 2, thermal imaging camera 100 includes adepressible trigger control 112 for capturing an infrared and visiblelight image, and buttons 116, which form part of the user interface, forcontrolling other aspects of the operation of the camera. A differentnumber or arrangement of user input media are possible, and it should beappreciated that the disclosure is not limited in this respect. Forexample, thermal imaging camera 100 may include a touch screen display108 which receives user input by depressing different portions of thescreen.

FIG. 3 is a functional block diagram illustrating components of anexample of thermal imaging camera 100. Thermal imaging camera 100includes an IR camera module 200, front end circuitry 202. The IR cameramodule 200 and front end circuitry 202 are sometimes referred to incombination as front end stage or front end components 204 of theinfrared camera 100. Thermal imaging camera 100 may also include avisible light camera module 206, a display 108, a user interface 208,and an output/control device 210.

Infrared camera module 200 may be configured to receive infrared energyemitted by a target scene and to focus the infrared energy on aninfrared sensor for generation of infrared energy data, e.g., that canbe displayed in the form of an infrared image on display 108 and/orstored in memory. Infrared camera module 200 can include any suitablecomponents for performing the functions attributed to the module herein.In the example of FIG. 3, infrared camera module 200 is illustrated asincluding infrared lens assembly 104 and infrared sensor 220. Asdescribed above with respect to FIGS. 1 and 2, infrared lens assembly104 includes at least one lens that takes infrared energy emitted by atarget scene and focuses the infrared energy on infrared sensor 220.Infrared sensor 220 responds to the focused infrared energy bygenerating an electrical signal that can be converted and displayed asan infrared image on display 108.

Infrared sensor 220 may include one or more focal plane arrays (FPA)that generate electrical signals in response to infrared energy receivedthrough infrared lens assembly 104. Each FPA can include a plurality ofinfrared sensor elements including, e.g., bolometers, photon detectors,or other suitable infrared sensor elements. In operation, each sensorelement, which may each be referred to as a sensor pixel, may change anelectrical characteristic (e.g., voltage or resistance) in response toabsorbing infrared energy received from a target scene. In turn, thechange in electrical characteristic can provide an electrical signalthat can be received by a processor 222 and processed into an infraredimage displayed on display 108.

For instance, in examples in which infrared sensor 220 includes aplurality of bolometers, each bolometer may absorb infrared energyfocused through infrared lens assembly 104 and increase in temperaturein response to the absorbed energy. The electrical resistance of eachbolometer may change as the temperature of the bolometer changes. Witheach detector element functioning as a sensor pixel, a two-dimensionalimage or picture representation of the infrared radiation can be furthergenerated by translating the changes in resistance of each detectorelement into a time-multiplexed electrical signal that can be processedfor visualization on a display or storage in memory (e.g., of acomputer). Processor 222 may measure the change in resistance of eachbolometer by applying a current (or voltage) to each bolometer andmeasure the resulting voltage (or current) across the bolometer. Basedon these data, processor 222 can determine the amount of infrared energyemitted by different portions of a target scene and control display 108to display a thermal image of the target scene.

Independent of the specific type of infrared sensor elements included inthe FPA of infrared sensor 220, the FPA array can define any suitablesize and shape. In some examples, infrared sensor 220 includes aplurality of infrared sensor elements arranged in a grid pattern suchas, e.g., an array of sensor elements arranged in vertical columns andhorizontal rows. In various examples, infrared sensor 220 may include anarray of vertical columns by horizontal rows of, e.g., 16×16, 50×50,160×120, 120×160, or 650×480. In other examples, infrared sensor 220 mayinclude a smaller number of vertical columns and horizontal rows (e.g.,1−1), a larger number vertical columns and horizontal rows (e.g.,1000×1000), or a different ratio of columns to rows.

In certain embodiments a Read Out Integrated Circuit (ROIC) isincorporated on the IR sensor 220. The ROIC is used to output signalscorresponding to each of the sensor pixels. Such ROIC is commonlyfabricated as an integrated circuit on a silicon substrate. Theplurality of detector elements may be fabricated on top of the ROIC,wherein their combination provides for the IR sensor 220. In someembodiments, the ROIC can include components discussed elsewhere in thisdisclosure (e.g. an analog-to-digital converter (ADC)) incorporateddirectly onto the FPA circuitry. Such integration of the ROIC, or otherfurther levels of integration not explicitly discussed, should beconsidered within the scope of this disclosure.

As described above, the IR sensor 220 generates a series of electricalsignals corresponding to the infrared radiation received by eachinfrared detector element to represent a thermal image. A “frame” ofthermal image data is generated when the voltage signal from eachinfrared detector element is obtained by scanning all of the rows thatmake up the IR sensor 220. Again, in certain embodiments involvingbolometers as the infrared detector elements, such scanning is done byswitching a corresponding detector element into the system circuit andapplying a bias voltage across such switched-in element. Successiveframes of thermal image data are generated by repeatedly scanning therows of the IR sensor 220, with such frames being produced at a ratesufficient to generate a video representation (e.g. 30 Hz, or 60 Hz) ofthe thermal image data.

The front end circuitry 202 includes circuitry for interfacing with andcontrolling the IR camera module 200. In addition, the front endcircuitry 202 initially processes and transmits collected infrared imagedata to a processor 222 via a connection therebetween. Morespecifically, the signals generated by the IR sensor 220 are initiallyconditioned by the front end circuitry 202 of the thermal imaging camera100. In certain embodiments, as shown, the front end circuitry 202includes a bias generator 224 and a pre-amp/integrator 226. In additionto providing the detector bias, the bias generator 224 can optionallyadd or subtract an average bias current from the total current generatedfor each switched-in detector element. The average bias current can bechanged in order (i) to compensate for deviations to the entire array ofresistances of the detector elements resulting from changes in ambienttemperatures inside the thermal imaging camera 100 and (ii) tocompensate for array-to-array variations in the average detectorelements of the IR sensor 220. Such bias compensation can beautomatically controlled by the thermal imaging camera 100 or software,or can be user controlled via input to the output/control device 210 orprocessor 222. Following provision of the detector bias and optionalsubtraction or addition of the average bias current, the signals can bepassed through a pre-amp/integrator 226. Typically, thepre-amp/integrator 226 is used to condition incoming signals, e.g.,prior to their digitization. As a result, the incoming signals can beadjusted to a form that enables more effective interpretation of thesignals, and in turn, can lead to more effective resolution of thecreated image. Subsequently, the conditioned signals are sent downstreaminto the processor 222 of the thermal imaging camera 100.

In some embodiments, the front end circuitry 202 can include one or moreadditional elements for example, additional sensors 228 or an ADC 230.Additional sensors 228 can include, for example, temperature sensors,visual light sensors (such as a CCD), pressure sensors, magneticsensors, etc. Such sensors can provide additional calibration anddetection information to enhance the functionality of the thermalimaging camera 100. For example, temperature sensors can provide anambient temperature reading near the IR sensor 220 to assist inradiometry calculations. A magnetic sensor, such as a Hall Effectsensor, can be used in combination with a magnet mounted on the lens toprovide lens focus position information. Such information can be usefulfor calculating distances, or determining a parallax offset for use withvisual light scene data gathered from a visual light sensor.

An ADC 230 can provide the same function and operate in substantiallythe same manner as discussed below, however its inclusion in the frontend circuitry 202 may provide certain benefits, for example,digitization of scene and other sensor information prior to transmittalto the processor 222 via the connection therebetween. In someembodiments, the ADC 230 can be integrated into the ROIC, as discussedabove, thereby eliminating the need for a separately mounted andinstalled ADC 230.

In some embodiments, front end components can further include a shutter240. A shutter 240 can be externally or internally located relative tothe lens and operate to open or close the view provided by the IR lensassembly 104. As is known in the art, the shutter 240 can bemechanically positionable, or can be actuated by an electro-mechanicaldevice such as a DC motor or solenoid. Embodiments of the invention mayinclude a calibration or setup software implemented method or settingwhich utilize the shutter 240 to establish appropriate bias levels foreach detector element.

Components described as processors within thermal imaging camera 100,including processor 222, may be implemented as one or more processors,such as one or more microprocessors, digital signal processors (DSPs),application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), programmable logic circuitry, or the like, eitheralone or in any suitable combination. Processor 222 may also includememory that stores program instructions and related data that, whenexecuted by processor 222, cause thermal imaging camera 100 andprocessor 222 to perform the functions attributed to them in thisdisclosure. Memory may include any fixed or removable magnetic, optical,or electrical media, such as RAM, ROM, CD-ROM, hard or floppy magneticdisks, EEPROM, or the like. Memory may also include a removable memoryportion that may be used to provide memory updates or increases inmemory capacities. A removable memory may also allow image data to beeasily transferred to another computing device, or to be removed beforethermal imaging camera 100 is used in another application. Processor 222may also be implemented as a System on Chip that integrates allcomponents of a computer or other electronic system into a single chip.These elements manipulate the conditioned scene image data deliveredfrom the front end stages 204 in order to provide output scene data thatcan be displayed or stored for use by the user. Subsequently, theprocessor 222 (processing circuitry) sends the processed data to adisplay 108 or other output/control device 210.

During operation of thermal imaging camera 100, processor 222 cancontrol infrared camera module 200 to generate infrared image data forcreating an infrared image. Processor 222 can generate a digital “frame”of infrared image data. By generating a frame of infrared image data,processor 222 captures an infrared image of a target scene at a givenpoint in time.

Processor 222 can capture a single infrared image or “snap shot” of atarget scene by measuring the electrical signal of each infrared sensorelement included in the FPA of infrared sensor 220 a single time.Alternatively, processor 222 can capture a plurality of infrared imagesof a target scene by repeatedly measuring the electrical signal of eachinfrared sensor element included in the FPA of infrared sensor 220. Inexamples in which processor 222 repeatedly measures the electricalsignal of each infrared sensor element included in the FPA of infraredsensor 220, processor 222 may generate a dynamic thermal image (e.g., avideo representation) of a target scene. For example, processor 222 maymeasure the electrical signal of each infrared sensor element includedin the FPA at a rate sufficient to generate a video representation ofthermal image data such as, e.g., 30 Hz or 60 Hz. Processor 222 mayperform other operations in capturing an infrared image such assequentially actuating a shutter 240 to open and close an aperture ofinfrared lens assembly 104, or the like.

With each sensor element of infrared sensor 220 functioning as a sensorpixel, processor 222 can generate a two-dimensional image or picturerepresentation of the infrared radiation from a target scene bytranslating changes in an electrical characteristic (e.g., resistance)of each sensor element into a time-multiplexed electrical signal thatcan be processed, e.g., for visualization on display 108 and/or storagein memory. When displayed on a display 108, an infrared image cancomprise a plurality of display pixels. Display pixels can have anydefined relationship with corresponding sensor pixels. In some examples,each sensor pixel corresponds to a display pixel in an imagerepresentation of infrared data. In other examples, a plurality ofsensor pixels may be combined (e.g., averaged) to provide infraredinformation for a single display pixel. Because relationships betweendisplay pixels and sensor pixels are defined with respect to cameraoperation, the generic term “pixel” may refer to the sensor pixel, thedisplay pixel, or the data as it is processed from the sensor pixel tothe display pixel unless otherwise stated. Processor 222 may performcomputations to convert raw infrared image data into scene temperatures(radiometry) including, in some examples, colors corresponding to thescene temperatures.

Processor 222 may control display 108 to display at least a portion ofan infrared image of a captured target scene. In some examples,processor 222 controls display 108 so that the electrical response ofeach sensor element of infrared sensor 220 is associated with a singlepixel on display 108. In other examples, processor 222 may increase ordecrease the resolution of an infrared image so that there are more orfewer pixels displayed on display 108 than there are sensor elements ininfrared sensor 220. Processor 222 may control display 108 to display anentire infrared image (e.g., all portions of a target scene captured bythermal imaging camera 100) or less than an entire infrared image (e.g.,a lesser port of the entire target scene captured by thermal imagingcamera 100). Processor 222 may perform other image processing functions,as described in greater detail below.

Independent of the specific circuitry, thermal imaging camera 100 may beconfigured to manipulate data representative of a target scene so as toprovide an output that can be displayed, stored, transmitted, orotherwise utilized by a user.

Thermal imaging camera 100 includes visible light camera module 206.Visible light camera module 206 may be configured to receive visiblelight energy from a target scene and to focus the visible light energyon a visible light sensor for generation of visible light energy data,e.g., that can be displayed in the form of a visible light image ondisplay 108 and/or stored in memory. Visible light camera module 206 caninclude any suitable components for performing the functions attributedto the module herein. In the example of FIG. 3, visible light cameramodule 206 is illustrated as including visible light lens assembly 106and visible light sensor 242. As described above with respect to FIGS. 1and 2, visible light lens assembly 106 includes at least one lens thattakes visible light energy emitted by a target scene and focuses thevisible light energy on visible light sensor 242. Visible light sensor242 responds to the focused energy by generating an electrical signalthat can be converted and displayed as a visible light image on display108.

Visible light sensor 242 may include a plurality of visible light sensorelements such as, e.g., CMOS detectors, CCD detectors, PIN diodes,avalanche photo diodes, or the like. The number of visible light sensorelements may be the same as or different than the number of infraredlight sensor elements.

In operation, optical energy received from a target scene may passthrough visible light lens assembly 106 and be focused on visible lightsensor 242. When the optical energy impinges upon the visible lightsensor elements of visible light sensor 242, photons within thephotodetectors may be released and converted into a detection current.Processor 222 can process this detection current to form a visible lightimage of the target scene.

During use of thermal imaging camera 100, processor 222 can controlvisible light camera module 206 to generate visible light data from acaptured target scene for creating a visible light image. The visiblelight data may include luminosity data indicative of the color(s)associated with different portions of the captured target scene and/orthe magnitude of light associated with different portions of thecaptured target scene. Processor 222 can generate a “frame” of visiblelight image data by measuring the response of each visible light sensorelement of thermal imaging camera 100 a single time. By generating aframe of visible light data, processor 222 captures visible light imageof a target scene at a given point in time. Processor 222 may alsorepeatedly measure the response of each visible light sensor element ofthermal imaging camera 100 so as to generate a dynamic thermal image(e.g., a video representation) of a target scene, as described abovewith respect to infrared camera module 200.

With each sensor element of visible light camera module 206 functioningas a sensor pixel, processor 222 can generate a two-dimensional image orpicture representation of the visible light from a target scene bytranslating an electrical response of each sensor element into atime-multiplexed electrical signal that can be processed, e.g., forvisualization on display 108 and/or storage in memory.

Processor 222 may control display 108 to display at least a portion of avisible light image of a captured target scene. In some examples,processor 222 controls display 108 so that the electrical response ofeach sensor element of visible light camera module 206 is associatedwith a single pixel on display 108. In other examples, processor 222 mayincrease or decrease the resolution of a visible light image so thatthere are more or fewer pixels displayed on display 108 than there aresensor elements in visible light camera module 206. Processor 222 maycontrol display 108 to display an entire visible light image (e.g., allportions of a target scene captured by thermal imaging camera 100) orless than an entire visible light image (e.g., a lesser port of theentire target scene captured by thermal imaging camera 100).

In these and other examples, processor 222 may control display 108 toconcurrently display at least a portion of the visible light imagecaptured by thermal imaging camera 100 and at least a portion of theinfrared image captured by thermal imaging camera 100. Such a concurrentdisplay may be useful in that an operator may reference the featuresdisplayed in the visible light image to help understand the featuresconcurrently displayed in the infrared image, as the operator may moreeasily recognize and distinguish different real-world features in thevisible light image than the infrared image. In various examples,processor 222 may control display 108 to display the visible light imageand the infrared image in side-by-side arrangement, in apicture-in-picture arrangement, where one of the images surrounds theother of the images, or any other suitable arrangement where the visiblelight and the infrared image are concurrently displayed.

For example, processor 222 may control display 108 to display thevisible light image and the infrared image in a combined arrangement. Insuch an arrangement, for a pixel or set of pixels in the visible lightimage representative of a portion of the target scene, there exists acorresponding pixel or set of pixels in the infrared image,representative of substantially the same portion of the target scene.Similarly, images having corresponding pixels (i.e., pixelsrepresentative of the same portion of the target scene) can be referredto as corresponding images. Thus, in some such arrangements, thecorresponding visible light image and the infrared image may besuperimposed on top of one another, at corresponding pixels. An operatormay interact with user interface 208 to control the transparency oropaqueness of one or both of the images displayed on display 108. Forexample, the operator may interact with user interface 208 to adjust theinfrared image between being completely transparent and completelyopaque and also adjust the visible light image between being completelytransparent and completely opaque. Such an exemplary combinedarrangement, which may be referred to as an alpha-blended arrangement,may allow an operator to adjust display 108 to display an infrared-onlyimage, a visible light-only image, of any overlapping combination of thetwo images between the extremes of an infrared-only image and a visiblelight-only image. Processor 222 may also combine scene information withother data, such as radiometric data, alarm data, and the like. Ingeneral, an alpha-blended combination of visible light and infraredimages can comprise anywhere from 100 percent infrared and 0 percentvisible light to 0 percent infrared and 100 percent visible light. Insome embodiments, the amount of blending can be adjusted by a user ofthe camera. Thus, in some embodiments, a blended image can be adjustedbetween 100 percent visible light and 100 percent infrared.

Additionally, in some embodiments, the processor 222 can interpret andexecute commands from user interface 208, an output/control device 210.This can involve processing of various input signals and transferringthose signals to the front end circuitry 202 via a connectiontherebetween. Components (e.g. motors, or solenoids) proximate the frontend circuitry 202 can be actuated to accomplish the desired controlfunction. Exemplary control functions can include adjusting the focus,opening/closing a shutter, triggering sensor readings, adjusting biasvalues, etc. Moreover, input signals may be used to alter the processingof the image data that occurs in the processor 222.

Processor can further include other components to assist with theprocessing and control of the infrared imaging camera 100. For example,as discussed above, in some embodiments, an ADC can be incorporated intothe processor 222. In such a case, analog signals conditioned by thefront-end stages 204 are not digitized until reaching the processor 222.Moreover, some embodiments can include additional on board memory forstorage of processing command information and scene data, prior totransmission to the display 108 or the output/control device 210.

An operator may interact with thermal imaging camera 100 via userinterface 208, which may include buttons, keys, or another mechanism forreceiving input from a user. The operator may receive output fromthermal imaging camera 100 via display 108. Display 108 may beconfigured to display an infrared-image and/or a visible light image inany acceptable palette, or color scheme, and the palette may vary, e.g.,in response to user control. In some examples, display 108 is configuredto display an infrared image in a monochromatic palette such asgrayscale or amber. In other examples, display 108 is configured todisplay an infrared image in a color palette such as, e.g., ironbow,blue-red, or other high contrast color scheme. Combinations of grayscaleand color palette displays are also contemplated.

While processor 222 can control display 108 to concurrently display atleast a portion of an infrared image and at least a portion of a visiblelight image in any suitable arrangement, a picture-in-picturearrangement may help an operator to easily focus and/or interpret athermal image by displaying a corresponding visible image of the samescene in adjacent alignment.

A power supply (not shown) delivers operating power to the variouscomponents of thermal imaging camera 100 and, in some examples, mayinclude a rechargeable or non-rechargeable battery and a powergeneration circuit.

During operation of thermal imaging camera 100, processor 222 controlsinfrared camera module 200 and visible light camera module 206 with theaid of instructions associated with program information that is storedin memory to generate a visible light image and an infrared image of atarget scene. Processor 222 further controls display 108 to display thevisible light image and/or the infrared image generated by thermalimaging camera 100.

As noted, in some situations, it can be difficult to differentiatebetween real-world features of the target scene in a thermal image. Inaddition to supplementing the infrared image with visible lightinformation, in some embodiments, it can be useful to emphasize physicaledges within the target scene. While edge detection can be difficult toperform in an infrared image, known edge detection methods can beperformed on a corresponding visible light image of the same targetscene. Because of the corresponding relationship between the infraredimage and the visible light image, visible light pixels determined torepresent a physical edge in the target scene correspond to infraredpixels also representing the edge in the infrared image.

Exemplary edge detection methods can include Canny edge detection,Kirsch operators, the Sobel operator, the Prewitt operator and theRoberts cross operator. An alternative method is generally described inthe paper “Focused edge detection using color histogram” by Kashiwagi,(hereinafter “Kashiwagi”), which is incorporated herein by reference. Insuch a process, the pixels in a visible light image are broken down intoa 3-dimensional color histogram, wherein the number of pixelscorresponding to each particular color making up the visible light imageis analyzed. Colors can be defined and distinguished in any of a varietyof color spaces, such as RGB, YCbCr, or CMYK, and placed into binscorresponding to the number of occurring pixels comprising specificcolors. Utilizing this analysis, the least populated bins (that is, theleast frequently occurring colors) are deemed to be representative of anedge within the image. Thus, if a pixel is in one of the least populatedbins, the pixel is determined to be an edge pixel, that is, a pixelrepresentative of an edge.

It should be noted that various edge detection processes can detectboundaries present in a visible light image that may or may notrepresent true physical extent of an object in the target scene.Accordingly, as used herein, edges are used generally to describe anydetectable boundary in the visible light image that may not correspondexclusively to a physical extent of an object.

In some embodiments, edge pixels can be defined by a threshold appliedto bins or the pixels themselves. For example, pixels in a certainnumber of the least populated bins (e.g., 5 least populated bins) orpixels in the lowest percentile of populated bins (e.g., bottom 5% ofbin population) can be deemed to be edge pixels. In other examples, eachof the least populated bins is deemed to comprise edge pixels until thenumber or percentage of edge pixels crosses a certain threshold. In somesuch embodiments, a user can define the threshold in order to determinethe edge detection sensitivity. In further embodiments, the detectededge can be indicated to the user in real time, allowing the user todynamically adjust the edge detection sensitivity while simultaneouslyobserving the effect. In some embodiments, the edge detection method isemployed by the processor 222 while presenting the detected edge to theuser via display 108. The user can then adjust the edge detectionsensitivity via the user interface 208 while observing the effectreal-time on the display.

In an alternative edge detection process, each pixel can be analyzedwithout sampling the entire RGB composition of a visible light image.For example, in some embodiments, an M by N pixel kernel is defined andanalyzed, wherein M and N are any positive integers. In furtherembodiments, a square N by N pixel kernel is analyzed, and in stillfurther embodiments N is an odd integer and the target pixel is in thecenter of the N by N square. In such embodiments, edges can be detectedin a two-step process. First, the variance of a particular kernelincluding a target pixel is analyzed. If the variance exceeds a certainthreshold, the target pixel is compared to mean pixel value within thekernel. If the target pixel is a certain threshold above the mean of thekernel, the target pixel is deemed an edge pixel. In some embodiments,the compared values of pixels can comprise luminance values detected ineach pixel.

In this exemplary edge detection process, multiple thresholds exist foruser manipulation. In addition, each pixel need only be targeted andanalyzed once, and no subsequent processing or iteration need beperformed. As such, this process can potentially perform edge detectionfaster with respect to a captured frame. Faster edge detection canfacilitate real-time edge detection in portable cameras, such as ahand-held thermal imaging camera. In some embodiments, thermal imagingcamera can detect edges and indicate edge information to the user inreal time. In some examples, real time edge detection and indication canbe performed at a variety of frame rates, such as 30 or 60 frames persecond. Generally, any value can be used. In some embodiments, the framerate can be selectable by a user.

Various edge detection methods have been described. In general, withrespect to visible light and infrared images, visible light imagesprovide more distinct edge features detectable by such methods. However,as described, in some embodiments, a thermal imaging camera can acquirean infrared and a corresponding visible light image of a target scene,in which pixels of the infrared image correspond to pixels of thevisible light image and are representative of substantially the sameportion of the target scene. Thus, an edge detected in a visible lightimage can be used to detect a corresponding edge in an infrared imagedespite a possible lack of thermal differentiation across the edge.

It should be noted that in some embodiments, the pixel count and/ordensity of a visible light and a corresponding infrared image need notbe the same. In some embodiments, for example, there can be more visiblelight pixels in a single visible light frame than there are infraredpixels in the corresponding infrared frame, or vice versa. Thus,corresponding pixels need not have a direct one-to-one relationship.Rather, in some embodiments, a single infrared pixel has a plurality ofcorresponding visible light pixels, or a visible light pixel has aplurality of corresponding infrared pixels. Additionally oralternatively, in some embodiments, not all visible light pixels havecorresponding infrared pixels, or vice versa. Such embodiments may beindicative of, for example, a picture-in-picture type display aspreviously discussed. Thus, a visible light pixel will not necessarilyhave the same pixel coordinate within the visible light image as does acorresponding infrared pixel. Accordingly, as used herein, correspondingpixels generally refers pixels from any image (e.g., a visible lightimage, an infrared image, a combined image, a display image, etc.)comprising information from substantially the same portion of the targetscene. Such pixels need not have a one-to-one relationship betweenimages and need not have similar coordinate positions within theirrespective images.

Once pixels representing edges in the target scene have been detected,there are several ways in which the edge data can be incorporated into adisplayed image in order to assist a user in interpreting a thermalimage. FIG. 4 is a process-flow diagram illustrating a general methodfor incorporating detected edge information into a display imageincluding infrared data. The process can be performed, for example, by aprocessor. In general, a processor can receive 250 a visible light imageand a corresponding infrared image of a target scene. Receiving 250images can include, for example, capturing the images with a thermalimaging camera, receiving image data representative of a thermal scenedirectly from imaging arrays, or uploading existing images from memorystorage. The processor can detect edges 252 in the visible light imageby analyzing the visible light image data and using any appropriatemethod. Considerations such as the power and intended function of theprocessor (i.e., real-time video representation of a target scene,post-capture processing of images, etc.) may account for the edgedetection method to be used. In general, VL pixels corresponding todetected edges in the VL image data can be considered VL edge pixels. Insome examples, VL pixels not corresponding to detected edges in the VLimage data can be considered VL non-edge pixels, though such pixels neednot always be specified as such. In some embodiments, the set of pixelsthat are detected as edge pixels can define a first set of pixels in thevisible light image. Similarly, corresponding pixels in the infraredimage can define a first set of pixels in the infrared image, andcorresponding pixels in the display image can define a first set ofpixels in the display image.

The processor can then select 254 a pixel from the visible light imageand determine 256 if the pixel is an edge pixel and generate acorresponding display pixel accordingly. A corresponding display pixelis a pixel to be used in a resultant image that can be intended fordisplay. It is a corresponding pixel in that it comprises informationfrom substantially the same portion of the target scene as the selectedvisible light pixel and its corresponding infrared pixel. In someembodiments, if the selected pixel is an edge pixel, the processor cangenerate 258 a corresponding display pixel including a predeterminedcolor. That is, for display pixels corresponding to VL edge pixels, theprocessor can assign display image data to each such display pixel ofthe predetermined color to emphasize the location of edges in thedisplay image data. The predetermined color can be a color selected, forexample, by a user from a list or a color palette. It should be notedthat assigning display image data of the predetermined color can includethe color exclusively, or can comprise some blended or otherwisemodified version of the predetermined color.

If the selected pixel is not an edge pixel, the processor can generate260 a corresponding display pixel including information from thecorresponding infrared pixel. The display pixel including informationfrom the corresponding infrared pixel can include data from the infraredpixel exclusively, or could comprise a blend of the correspondinginfrared and visible light pixels. That is, for display pixelscorresponding to VL pixels that are the VL non-edge pixels, theprocessor can assign display image data to each such display pixel as ablend of the IR image data and the VL image data of the associated IRpixel and VL pixel corresponding to such display pixel. It should benoted that the blend of the IR and VL image data can range from 100percent IR image data and 0 percent VL image data to 0 percent IR imagedata and 100 percent VL image data. That is, the blend of the IR imagedata and the VL image data can comprise, for example, entire IR imagedata and no VL image data.

In general, a plurality of such display pixels can be generated in orderto generate a display image associated with such pixels. In variousembodiments, some of the display pixels include IR image data associatedwith corresponding IR pixels. Display pixels corresponding to VL edgepixels can include a predetermined color. In some embodiments, suchdisplay pixels do not include any IR image data associated withcorresponding IR pixels. Accordingly, in some configurations, displaypixels not corresponding to VL edge pixels can include IR image dataassociated with corresponding IR pixels, while display pixelscorresponding to VL edge pixels do not include such corresponding IRimage data.

In some embodiments, the processor can determine 262 if every pixel ofinterest has been analyzed. That is, the processor can determine ifevery pixel for which edge detection is desired has been analyzed. Ifnot, the processor selects 254 another visible light pixel and theanalysis repeats. If, however, each pixel of interest has been analyzed,the set of generated display pixels can be combined to generate adisplay image including infrared data and emphasized edge data at pixelscorresponding to non-edge and edge pixels in the visible light image,respectively. In some embodiments, the processor does not necessarilyperform the method in a pixel-by-pixel manner as illustrated. Forexample, once edge pixels are detected 252, each edge pixel can be usedto generate edge display pixels in the display image while each non-edgepixel can be used to generate non-edge display pixels in the displayimage simultaneously without requiring an iterative process asillustrated by the feedback arrow from 262. In some such embodiments,the processor need not explicitly check that each pixel of interest hasbeen analyzed. In addition, it should be noted that steps in theprocess-flow diagram of FIG. 4 may be permuted. For example, in someembodiments, the processor can analyze a single pixel in the visiblelight image, determine if that pixel is an edge pixel, and proceed togenerate the corresponding display pixel prior to analyzing anysubsequent pixels in the visible light image.

Various embodiments of the process illustrated in FIG. 4 are as follows:

Example 1: If a pixel is detected as an edge pixel in the visible lightimage, the corresponding display pixel in the display image will bedisplayed as a predetermined color, which can be selected by a user. Thedisplay pixels that do not corresponding to an edge pixel in the visiblelight image can be populated with corresponding infrared pixels from theinfrared image, or a blend of corresponding infrared and visible lightpixels. In instances in which blending is performed, in someembodiments, edge pixels detected in the visible light image will bereplaced with the predetermined color prior to blending. Subsequently,the visible light and infrared images are blended wherein any pixels inthe visible light image having exclusively the predetermined color willbe unaffected by blending and will be passed through the image blenderand to the display image as the predetermined color.

Example 2: If a pixel is detected as an edge pixel in the visible lightimage, the corresponding pixel in the infrared image is replaced by apredetermined color to create a modified infrared image. Subsequently,the modified infrared image can be presented as the display image, orcan be blended with the visible light image to create the display image.In some examples, blending can be performed such that the pixelscorresponding to the detected VL edge pixels in the resulting displayimage include the predetermined color, though not necessarily exactly asincluded in the modified IR image due to the blending step. However,such display pixels can still be said to include the predeterminedcolor.

In any such embodiment, the display image can be presented on a displayincluding infrared information and having detected edge pixels includingthe predetermined color. As mentioned, in some embodiments, some of thedisplay pixels include IR image data associated with corresponding IRpixels. In some such embodiments, however, display pixels correspondingto VL edge pixels include a predetermined color and do not include anyIR image data associated with corresponding IR pixels.

In generating such a display image, IR image information can begenerally presented across portions of the display image, while portionsof the display image corresponding to detected edges can be emphasizedwith the predetermined color. In particular, in some such embodiments,the display pixels corresponding to VL edge pixels do not include any IRimage data in order to more clearly distinguish such edge pixels fromnon-edge pixels that do include IR image data. Thus, the display imagecan include IR image data wherein detected edges are emphasized with apredetermined color, thereby providing better context of the targetscene to a viewer of the infrared image data. In addition, inembodiments, the replacing of pixels with the predetermined color priorto the processing image data and/or not requiring the inclusion of IRimage data in display pixels corresponding to VL edge pixels can reducethe number of processing steps and/or the processing power required forgenerating the display image. Accordingly, display images can begenerated quickly and presented in substantially real time.

In various embodiments, the display image can comprise a blend ofvisible light and infrared pixels, with the amount of blendingadjustable by a user. The display image can be presented in apicture-in-picture arrangement, wherein the display image is presentedwithin a visible light image, for example. In some embodiments, avisible light image can be presented in color surrounding a blendeddisplay image. In further embodiments, a portion of the visible lightimage to be blended is converted into grayscale prior to blending withthe corresponding portion of a palettized infrared image. The resultingblended portion of the display image can be presented within theremaining (unblended) visible light image, which can remain colored.That is, in some embodiments, only pixels to be blended with infraredpixels are converted to grayscale for blending, the remaining visiblelight pixels remain presented in color.

Such a picture-in-picture display can assist the user in differentiatingbetween elements of the target scene. In various embodiments in whichedges are detected and/or emphasized in a resulting display image, theedges are emphasized only within the blended portion of the displayimage (though the edge pixels themselves need not be blended).Alternatively, edge pixels in the display image can be presentedthroughout the entire display image, including outside of the blendedportion thereof.

Each of the various embodiments herein described can be carried out by aprocessor or series of processors in, for example, a thermal imagingcamera or an external device such as a PC. In addition, embodiments canbe performed on corresponding visible light and infrared images storedin memory, or can be performed in situ on received visible light andinfrared image data in order to execute the method and present resultingdisplay images in substantially real time on a display. Employed methodscan be selected, for example, based on desired functionality, availableand required processing power/speed, or other reasons. In addition,methods can be carried out in any appropriate systems for processingand/or displaying image data, such as various mobile devices includingsmart phones and tablets.

With further reference to FIG. 3, in one exemplary embodiment, a thermalimaging camera comprises a processor 222 such as an FPGA for processingand blending visible light and infrared images as described above, andcomprises a second processor 223 (e.g., a second FPGA) included in orotherwise in communication with the visible light module 206. The secondprocessor 223 can receive and process visible light image data from thevisible light module 206 and detect edge pixels in the visible lightimage. In some embodiments, the second processor 223 processes thevisible light image data by detecting edge pixels and replacing thedetected edge pixels with a predetermined color prior to passing themodified visible light image data to the processor 222. Thepredetermined color can be selected by a user via the user interface 208and directed to the second processor 223 for visible light image datamodification.

The processor 222 can receive modified visible light image data andproceed according to any of several methods to incorporate edge datainto a display image. For example, the processor 222 can be configuredto blend at least portions of the modified visible light data from thesecond processor 223 with corresponding portions of infrared image datafrom the infrared camera module 200 or front end circuitry 202 to createa display image. Blending can range from 0 to 100 percent visible lightand 100 to 0 percent infrared, and can be determined by a user. In someembodiments, the blending process is such that if the processor 222detects a pixel in the modified visible light image data havingexclusively the predetermined color, that pixel is unaffected by theblending process and the corresponding pixel in the display imageconsists of only the predetermined color. Similarly, in someembodiments, prior to the blending, pixels in the display imagecorresponding to detected VL edge pixels can be assigned thepredetermined color, and a blending process for generating such displaypixels can be skipped entirely. In a similar embodiment, display pixelscorresponding to VL edge pixels can be replaced by pixels of thepredetermined color. Thus, in at least a portion of the display image,detected edges are emphasized to the user in the predetermined color. Insome cases, a portion of the display image comprises infrared image datain the non-edge pixels and the predetermined color in the edge pixels.

FIG. 5 is a process-flow diagram illustrating exemplary operation ofgenerating and modifying a display image. In various embodiments, a usercan initiate 270 edge detection in a visible light image having acorresponding infrared image. The user can be the user of a thermalimaging camera or other computing device either with or without theability to acquire visible and/or infrared images. In various devices,the images can be acquired by the device or otherwise stored in thedevice in memory, for example. In initiating 270 edge detection, thedevice can perform any appropriate method for detecting edges in thevisible light image.

The user can select 272 a color for use in emphasizing the detectededges in a display image. In some embodiments, the user can select thecolor from a predefined list of colors. In other embodiments, the usercan select any color in RGB space by inputting individual R (red) G(green) and B (blue) values, or by selecting a color from a colorpalette. Similarly, among various embodiments, the user can define acolor by inputting values into any known color space, such as YCbCr andCMYK, for example. The selected color can be defined as a predeterminedcolor and used to display the location of detected edges in the displayimage either directly, or by blending the predetermined color withcorresponding pixel data, in some examples from the visible light image.

Beyond selecting 272 the predetermined color for emphasizing detectededges in the display image, the user can define 274 display imageparameters. Such parameters can include, for example, the desired colorpalette used to represent the infrared image data, the type of displayimage to be used (such as a picture-in-picture display, a blended image,or other known display types), and the amount of desired blendingbetween the visible light and infrared images. In some embodiments, theuser can also define the sensitivity of the edge detection. This canaffect the thickness of the edges presented in the display image as wellas the number of edges detected. In other embodiments, a defaultsensitivity is used.

The user can initiate 276 generation of the display image on the devicein various ways. In some devices, the user selects a visible light imageand a corresponding infrared image from which to produce a displayimage. Such might be the case in, for example, a PC. In other devices,such as a portable thermal imaging camera, the user may simply point thecamera at a target scene and display images will be generated in realtime. In other embodiments, the user can trigger image capture using thethermal imaging camera in order to acquire one or more frames of visiblelight and corresponding infrared images. Upon display image generation,the user can be presented with a display image comprising infrared imagedata from a target scene and emphasized edge data including thepredetermined color.

As previously described, the display image pixels corresponding to thedetected edges in the visible light image can, in some embodiments,consist exclusively of the predetermined color. In other embodiments,such pixels comprise the predetermined color blended with the visiblelight pixel, for example. In some configurations and with some devices,the nature of emphasizing the edge pixels can be selected by the userwhile defining 274 display image parameters. In other devices orparticular modes of operation, only a particular method of emphasizingdetected edges may be utilized. For example, during real time imageacquisition, edge detection, and display image display, some methods ofedge detection and/or display may require too much processing time toexecute while providing a real-time display.

In some configurations, a user can adjust 278 various parameters tomodify the display image. For example, in some embodiments, a user canadjust the type of visible light and infrared display (e.g., blend,picture-in-picture, etc.), the amount of blending in the blended portionof the image, the selected color or the edge detection sensitivity.Thus, a user can adjust parameters in order to change the presentationof the display image according to his or her preferences or correctpotential misrepresentation of the image (e.g., too many or not enoughedges are detected). In various embodiments, the user can capture,print, save, modify or monitor the display image in accordance with thetask being performed by the user.

It will be appreciated that various steps in the process illustrated inFIG. 5 can be combined, permuted and/or omitted. For example, initiating270 edge detection and initiating 276 generation of the display imagecan be included in the same step or performed simultaneously. A user canselect 272 a color (for example from a predefined list of colors, aninteractive color palette, or an input for a user to define values inany of a number of color spaces) to be the predetermined color, define274 the display image parameters, and initiate 276 generation of thedisplay image wherein the display image is already configured tocomprise emphasized edge data including the predetermined color. In sucha case, edge detection can be initiated 270 upon initiation 276 of thedisplay image generation.

In addition, the process outlined from a user's perspective in FIG. 5can be performed from the perspective of, for example, a processor,thermal imaging camera, stand-alone computer, or the like. Such aprocess could comprise receiving a command initiating edge detection,receiving a selected color, generating display image data and receivingcommands for adjusting the display image data. Additional exemplarysteps could include presenting display image data on a display,receiving VL image data and receiving IR image data. Receiving commandsfor adjusting at least one parameter can include commands for adjustingat least one of an edge detection threshold, the selected color, animage display type (e.g., picture-in-picture, overlap, etc.), and anamount of blending to incorporate between VL and IR image data. Asnoted, various steps in the process can be omitted and/or permuted amongembodiments of the invention. For example, the processor can simplygenerate a display image according to various methods described hereinwithout receiving input from the user. Such an example can be carried inthe generation and presentation of a real-time display image includinginfrared image data and emphasized edges including a predeterminedcolor.

FIGS. 6A-6F illustrate various display images of a target scene. FIG. 6Ais a display image comprising the visible light image data of the targetscene. FIG. 6B is a display image comprising the infrared image data ofthe target scene. While objects of different temperatures are relativelydistinguishable within the scene, some edges noticeable in the visiblelight image (particularly edges showing details of portions of the imagesuch as stripes on the shoe) are obscured because of relatively uniformthermal radiation emitted from either side of the edge. Edges of thevisible light image can be detected using any appropriate edge detectionscheme such as those herein described. Moreover, in some embodiments,the sensitivity of a particular edge detection scheme can be adjusted bya user, resulting in more or fewer detected edges. FIG. 6C is anexemplary set of edge pixels detected in the visible light image of FIG.6A using an edge detection scheme with a first sensitivity.

FIG. 6D is a display image in which edges have been detected using anedge detection process and a first sensitivity, as illustrated in FIG.6C. As shown, edges from the infrared image of FIG. 6B are emphasizedwith the detected lines shown in FIG. 6C. Such lines provide context toa viewer viewing the display image. The display image of FIG. 6D can bepresented as a blended image in which the visible light image data andcorresponding infrared image data are blended to produce the image. Asdiscussed, blending can range from 100 percent infrared and 0 percentvisible light to 0 percent infrared and 100 percent visible light. Theedge detection and display such as that shown in FIGS. 6C and 6D can beperformed using any of the methods and/or embodiments herein described.For example, the display image of FIG. 6D can be presented on a displayof a thermal imaging camera in real-time, continually updating as thecamera is directed at different target scenes and/or the target scenechanges. As discussed, in some examples, parameters of the display imageare adjustable by a user, such as the sensitivity of the edge detection.

FIG. 6E is a display image similar to that of FIG. 6C, in which detectededges in the image of FIGS. 6A and 6B are presented. In FIG. 6E, thedisplayed edges are detected using a second edge sensitivity, resultingin a greater number of detected edge pixels when compared to the firstedge sensitivity. For example, in the edge detection process used todetect the edge pixels of FIG. 6E, a detection threshold might be lowerthan for the edge detection process used to detect the edge pixelsrepresented in FIG. 6C.

FIG. 6F is a display image such as that shown in FIG. 6B, including theemphasized edge pixels of FIG. 6E. In the display image of FIG. 6F, manymore emphasized edge pixels are presented than in the same scene of FIG.6D. In this particular example, the sensitivity of the edge detectionprocess has been increased, thereby increasing the number of pixels inthe visible light image considered to be visible light edge pixels andpresented as such in the associated display image data. In someembodiments, the user can adjust the edge detection sensitivity in andobserve changes in the detected edges of a captured image in real time.In further embodiments, the edge detection sensitivity can be adjustedwhile the display image is updated in real time, allowing the user toimmediately observe the change in the detected edges while performing animaging operation.

Example thermal image cameras and related techniques have beendescribed. The techniques described in this disclosure may also beembodied or encoded in a computer-readable medium, such as anon-transitory computer-readable storage medium containing instructions.Instructions embedded or encoded in a computer-readable storage mediummay cause a programmable processor, or other processor, to perform themethod, e.g., when the instructions are executed. Computer readablestorage media may include random access memory (RAM), read only memory(ROM), a hard disk, optical media, or other computer readable media.

For example, an external computer comprising such computer readablemedium can receive corresponding visible light and infrared images froma thermal imaging camera or from memory and perform edge detectionand/or generate display images as described herein. In some embodiments,various portions of the techniques can be embodied in multiplecomponents. For example, a thermal imaging camera can detect edges in avisible light image and pass detected edge information to an externalcomputing device for generating the display image utilizing the detectedimages.

In further examples, embodiments of the invention can be embodied in adisplay system. The display system can be configured to receive VL andIR image data and carry out processes such as those herein described.Exemplary display systems can include one or more processors, a displayand a user interface for carrying out such processes. A display systemcan be incorporated into any appropriate device or system capable ofreceiving and processing image data. In some embodiments, the displaysystem can include a thermal imaging camera, such as a hand-held thermalimaging camera as described elsewhere herein in order to capturecorresponding VL and IR images and provide VL and IR image data to othercomponents of the imaging system. In further embodiments, the imagingsystem is fully incorporated into such a camera, or can consistessentially of a camera capable of carrying out any of the variousprocesses described.

Various embodiments have been described. Such examples are non-limiting,and do not define or limit the scope of the invention in any way.Rather, these and other examples are within the scope of the followingclaims.

1. A non-transitory computer readable medium containing executableinstructions causing one or more processors to perform a method ofcreating an infrared image of a scene that emphasizes the locations ofvisible edges, the method comprising: receiving visible light (VL) imagedata associated with a plurality of VL pixels indicative of a VL imageof a target scene; receiving infrared (IR) image data associated with aplurality of IR pixels indicative of an IR image of the target scene,each IR pixel having a corresponding one or more VL pixels indicative ofthe same portion of the target scene; determining which VL pixelscorrespond to edges in the VL image data and considering such pixels asVL edge pixels; and generating display image data associated with aplurality of display pixels, each display pixel having corresponding VLand IR pixels that correspond to the same portion of the target scene assuch display pixel; wherein some of the display pixels include IR imagedata associated with corresponding IR pixels, and display pixelscorresponding to VL edge pixels include a predetermined color and do notinclude IR image data associated with corresponding IR pixels in orderto emphasize the location of edges in the display image data.
 2. Thenon-transitory computer-readable medium of claim 1, wherein thepredetermined color is selectable by a user.
 3. The non-transitorycomputer-readable medium of claim 1, wherein, for display pixelscorresponding to VL pixels that are the VL non-edge pixels, thegenerated display image data of each such display pixel comprises ablend of between 0% and 100% IR image data from the associated IR pixelcorresponding to such display pixel.
 4. The non-transitorycomputer-readable medium of claim 3, wherein the amount of blending isselectable by a user.
 5. The non-transitory computer-readable medium ofclaim 1, further including, for IR pixels corresponding to the VL edgepixels, replacing the IR image data of each such IR pixel with thepredetermined color, creating modified IR image data, and wherein, fordisplay pixels corresponding to the VL edge pixels, including thepredetermined color includes, for each such display pixel, blending themodified IR image data associated with the corresponding IR pixel andthe corresponding VL image data.
 6. The non-transitory computer-readablemedium of claim 1, further configured to receive visible light andcorresponding infrared image data and generate a display image data insubstantially real time.
 7. The non-transitory computer-readable mediumof claim 6, wherein generating a display image data in substantiallyreal time comprises generating a display video comprising a frame rateof approximately 60 frames per second.
 8. The non-transitorycomputer-readable medium of claim 1, wherein determining which VL pixelscorrespond to the edges in the VL image data of the target scene: (i)measuring the variance in a kernel of VL pixels; and (ii) if thevariance is above a predetermined threshold value, comparing a targetpixel within the kernel to the mean pixel within the kernel.
 9. Adisplay system for generating and displaying a display image comprising:a processor; a display; and wherein the display system is configured toreceive a visible light (VL) image and a corresponding infrared (IR)image; detect pixels in the VL image representative of detected edges inthe VL image and consider such pixels as VL edge pixels; and generate adisplay image corresponding to the VL and IR images and comprising aplurality of display pixels, wherein at least one display pixelcorresponding to a VL pixel not considered to be a VL edge pixelcomprises corresponding IR image data; and display pixels correspondingto the VL pixels that are considered to be VL edge pixels comprise apredetermined color and do not comprise corresponding IR image data inorder to emphasize the location of edges in the display image data. 10.The system of claim 9, further comprising a portable, hand-held thermalimaging camera including: an infrared (IR) camera module comprising anIR lens assembly and an associated IR sensor for detecting IR images ofa target scene; and a visible light (VL) camera module comprising a VLlens assembly having an associated VL sensor for detecting VL images ofthe target scene; the system being further configured to: capture the VLimage of the target scene via the VL camera module; and capture thecorresponding IR image of a target scene via the IR camera module. 11.The system of claim 10, wherein the visible light camera modulecomprises a second processor, and wherein the second processor isconfigured to detect pixels in the VL image representative of detectededges in the VL image and considering such pixels as VL edge pixels andconsidering the VL pixels not corresponding to the detected edges as VLnon-edge pixels.
 12. The system of claim 11, wherein the secondprocessor is configured to replace the pixels in the VL image consideredto be VL edge pixels with pixels of exclusively the predetermined color,creating a modified VL image.
 13. The system of claim 12, wherein theprocessor is configured to: (i) receive the modified VL image from thesecond processor; and (ii) blend at least a portion of the modified VLimage with a corresponding portion of the IR image to generate a portionof the display image; wherein the pixels in the modified VL image havingexclusively the predetermined color are unaffected by the blending. 14.The system of claim 13, wherein the display image forms a part of apicture-in-picture image, wherein a first portion of thepicture-in-picture image comprises the display image; and a secondportion of the picture-in-picture image comprises an unblended portionof the VL image.
 15. The system of claim 9, further configured toreceive VL and IR images and generate display images in substantiallyreal time; and present the display images in substantially real time ona display.
 16. The system of claim 9, wherein detecting pixels in the VLimage representative of detected edges in the VL image comprisesexecuting an algorithm for detecting edges in the VL image; and whereinthe algorithm is adjustable by the user via a user interface such thatthe sensitivity of the edge detection can be adjusted by a user.
 17. Thesystem of claim 16, wherein the algorithm comprises: (i) measuring thevariance in a kernel of pixels; and (ii) if the variance is above apredetermined threshold value, comparing a target pixel within thekernel to the mean pixel within the kernel.
 18. The system of claim 16,wherein the algorithm comprises generating a color histogram of the VLpixels in the VL image and detecting VL edge pixels based on auser-defined threshold of color frequency in the color histogram. 19.The system of claim 9, wherein generating the display image comprisesreplacing pixels in the IR image corresponding to the VL pixelsconsidered to be VL edge pixels with the predetermined color.
 20. Amethod of enhancing infrared images comprising: receiving visible light(VL) image data associated with a plurality of VL pixels representativeof a target scene; receiving infrared (IR) image data associated with aplurality of IR pixels representative of at least a portion of thetarget scene, and at least a portion of the IR pixels corresponding toone or more VL pixels; detecting edge pixels in the VL image datarepresentative of edges in the VL image data; and generating displayimage data associated with a plurality of display pixels, wherein atleast one display pixel includes IR image data from at least onecorresponding IR pixel, and display pixels corresponding to edge pixelsin the VL image include a predetermined color and do not includecorresponding IR image data in order to emphasize the location of edgesin the display image data.